Evaluating the effectiveness of data quality framework in software engineering

The quality of data is important in research working with data sets because poor data quality may lead to invalid results. Data sets contain measurements that are associated with metrics and entities; however, in some data sets, it is not always clear which entities have been measured and exactly which metrics have been used. This means that measurements could be misinterpreted. In this study, we develop a framework for data quality assessment that determines whether a data set has sufficient information to support the correct interpretation of data for analysis in empirical research. The framework incorporates a dataset metamodel and a quality assessment process to evaluate the data set quality. To evaluate the effectiveness of our framework, we conducted a user study. We used observations, a questionnaire and think aloud approach to provide insights into the framework through participant thought processes while applying the framework. The results of our study provide evidence that most participants successfully applied the definitions of dataset category elements and the formal definitions of data quality issues to the datasets. Further work is needed to reproduce our results with more participants, and to determine whether the data quality framework is generalizable to other types of data sets

How Usability Defects Defer from Non-Usability Defects? : A Case Study on Open Source Projects

Usability is one of the software qualities attributes that is subjective and often considered as a less critical defect to be fixed. One of the reasons was due to the vague defect descriptions that could not convince developers about the validity of usability issues. Producing a comprehensive usability defect description can be a challenging task, especially in reporting relevant and important information. Prior research in improving defect report comprehension has often focused on defects in general or studied various aspects of software quality improvement such as triaging defect reports, metrics and predictions, automatic defect detection and fixing.  In this paper, we studied 2241 usability and non-usability defects from three open-source projects - Mozilla Thunderbird, Firefox for Android, and Eclipse Platform. We examined the presence of eight defect attributes - steps to reproduce, impact, software context, expected output, actual output, assume cause, solution proposal, and supplementary information, and used various statistical tests to answer the research questions. In general, we found that usability defects are resolved slower than non-usability defects, even for non-usability defect reports that have less information. In terms of defect report content, usability defects often contain output details and software context while non-usability defects are preferably explained using supplementary information, such as stack traces and error logs. Our research findings extend the body of knowledge of software defect reporting, especially in understanding the characteristics of usability defects. The promising results also may be valuable to improve software development practitioners' practice

Understanding usability defect reporting in software defect repositories

Software defect management is a critical component of good software engineering practice. The information reported about a defect is a key element to ensure defects are rectified effectively. However, based on research, reporting usability defects using an existing defect tracking system (DTS) is impractical. This is due to text-centric design and lack of features to support usability attributes. In addition, not all defects can be explained textually; especially defects that involve interface redesign. Another aspect to consider is that the reporters describe usability defects based on their usability knowledge and the information available at the time the defects are found. Defects stored in a DTS in a universal format. Therefore, when reporting usability defects there are some possibilities: the data may not be relevant or irrelevant, useful or not useful, or may even be beyond the reporter's knowledge. This makes it impossible to submit a high quality defect report. To address these issues, I propose a custom defect template that can adjust defect form according to whom, when and how the defect is found. In this way, it will provide flexibility to the reporters to record data based on their expertise and knowledge

Reporting usability defects-do reporters report what software developers need?

Reporting usability defects can be a challenging task, especially in convincing the software developers that the reported defect actually requires attention. Stronger evidence in the form of specific details is often needed. However, research to date in software defect reporting has not investigated the value of capturing different information based on defect type. We surveyed practitioners in both open source communities and industrial software organizations about their usability defect reporting practices to better understand information needs to address usability defect reporting issues. Our analysis of 147 responses show that reporters often provide observed result, expected result and steps to reproduce when describing usability defects, similar to the way other types of defects are reported. However, reporters rarely provide usability-related information. In fact, reporters ranked cause of the problem is the most difficult information to provide followed by usability principle, video recoding, UI event trace and title. Conversely, software developers consider cause of the problem as the most helpful information for them to fix usability defects. Our statistical analysis reveals a substantial gap between what reporters provide and what software developers need when fixing usability defects. We propose some remedies to resolve this gap

Reporting usability defects: Limitations of open source defect repositories and suggestions for improvement

Capturing good quality defect reports is critical for ensuring quick resolution of defects. While a range of studies into defect reporting exists, to date, very few focused on usability defect reporting issues can be found. To better understand issues specific to usability defect reporting, we carried out a survey of 56 contributors to the Mozilla and Google Chromium software projects spanning both usability defect reporters and developers. We discovered a range of limitations and key issues in current usability defect reporting tools and approaches identified as important by reporters and developers. In addition, we highlight opportunities to improve defect-reporting tools based on these open source communities' needs

ArguLens: Anatomy of Community Opinions On Usability Issues Using Argumentation Models

In open-source software (OSS), the design of usability is often influenced by the discussions among community members on platforms such as issue tracking systems (ITSs). However, digesting the rich information embedded in issue discussions can be a major challenge due to the vast number and diversity of the comments. We propose and evaluate ArguLens, a conceptual framework and automated technique leveraging an argumentation model to support effective understanding and consolidation of community opinions in ITSs. Through content analysis, we anatomized highly discussed usability issues from a large, active OSS project, into their argumentation components and standpoints. We then experimented with supervised machine learning techniques for automated argument extraction. Finally, through a study with experienced ITS users, we show that the information provided by ArguLens supported the digestion of usability-related opinions and facilitated the review of lengthy issues. ArguLens provides the direction of designing valuable tools for high-level reasoning and effective discussion about usability